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(57) Abstract: A system and method for 
ink database searching using handwriting 
feature synthesis is disclosed which 
allows a digital ink database to be 
searched using a text-based query. Using a 
writer-specific handwriting model derived 
from a handwriting recognition system or 
suitable training procedure, a text query 
is converted into feature vectors that are 
similar to the feature vectors that would 
have been extracted had the author of the 
digital ink database written the text query 
by hand. The feature vectors are then used 
to search the database. This allows the 
searching of a digital ink database when 
the only input mechanism available is text 
entry, and can allow a person other than the 
author of the digital ink database to search 
the digital ink database. 
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Digital Ink Database Searching Using Handwriting Feature 

Synthesis 

Technical Field 

5 The present invention broadly relates to digital processor implemented handwriting 
searching or recognition systems, and in particular, to a method of and apparatus 
for searching of a digital ink database using handwriting feature synthesis from a 
search query in text form. 

io Background Art 

"Digital ink database" as used herein refers to a database which stores handwritten 
characters, for example a string of handwritten characters forming a handwritten letter. 

15 Overview 

Pen-based computing systems provide a convenient and flexible means of human- 
computer interaction. Most people are very familiar with using pen and paper. This 
familiarity is exploited by known systems which use a pen-like device as a data 
entry and recording mechanism for text, drawings or calculations which are quite 
20 naturally supported by this medium. Additionally, written ink is a more expressive 
format than digital text, and ink-based systems can be language-independent. 

The increasing use of pen computing and the emergence of paper-based interfaces 
to networked computing resources (for example see: P. Lapstun, Netpage System 

25 Overview, Silverbrook Research Pty Ltd, 6 th June, 2000; and, Anoto, "Anoto, 
Ericsson, and Time Manager Take Pen and Paper into the Digital Age with the 
Anoto Technology", Press Release, 6 th April, 2000), has highlighted the need for 
techniques which are able to store, index, and search (raw) digital ink. Pen-based 
computing allows users to store data in the form of notes and annotations, and 

30 subsequently search this data based on hand-drawn queries. However, searching 
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handwritten text is more difficult than traditional text (e.g. ASCII text) searching 
due to inconsistencies in the production of handwriting and the stylistic variations 
between writers. 

5 Digital Ink Database Searching 

The traditional method of searching handwritten data in a digital ink database is to 
first convert the digital ink database and corresponding search query to standard 
text using pattern recognition techniques, and then to match the query text with the 
converted standard text in the database. Fuzzy text searching methods have been 

10 described, see P. Hall and G. Dowling, "Approximate String Matching", 
Computing Surveys, 12(4), pp. 381-402,1980, that perform text matching in the 
presence of character errors, similar to those produced by handwriting recognition 
systems. 

15 However, handwriting recognition accuracy remains low, and the number of errors 
introduced by handwriting recognition (both for the database entries and for the 
handwritten query) means that this technique does not work well. The process of 
converting handwritten information into text results in the loss of a significant 
amount of information regarding the general shape and dynamic properties of the 

20 handwriting. For example, some letters (e.g. 'u' and 'v\ 'v' and 'r\ 'f and 't\ 
etc.) are handwritten with a great deal of similarity in shape. Additionally, in many 
handwriting styles (particularly cursive writing), the identification of individual 
characters is highly ambiguous. 

25 Various techniques for directly searching and indexing a digital ink database are 
known in the prior art, see for example: A. Poon, K. Weber, and T.Cass, 
"Scribbler: A Tool for Searching Digital Ink", Proceedings of the ACM Computer- 
Human Interaction, pp. 58-64, 1994; I. Kamel, "Fast Retrieval of Cursive 
Handwriting", Proceedings of the 5 th International Conference on Information and 

30 Knowledge Management, Rockville, MD USA, November 12-16, 1996; W. Aref, 
D. Barbera, P. Vallabhaneni, "The Handwritten Trie: Indexing Electronic Ink", The 
1995 ACM SIGMOD International Conference on Management of Data, San Jose, 
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California, May 1995; W, Aref, D. Barbera, D. Lopresti, and A. Tomkins, "Ink as a 
First-Class Datatype in Multimedia Databases", Database System - Issues and 
Research Direction, pp. 113-163, 1996; and, R. Manmatha, C. Han, E. Riseman, 
and W. Croft, "Indexing Handwriting Using Word Matching", Proceedings of the 
5 First ACM International Conference on Digital Libraries, pp. 151-159, 1996. 

These systems use a similarity measure to compare a feature vector derived from a 
set of query pen strokes with a database of feature vectors derived from the digital 
ink database. The entries in the database that exhibit the greatest degree of 

10 similarity with the query are returned as matches. Additionally, some approaches 
create an index or use a partitioning scheme to avoid a sequential search of all 
entries in the database. See for example: D. Barbara, W. Aref, I. Kamel, and P. 
Vallabhaneni, "Method and Apparatus for Indexing a Plurality of Handwritten 
Objects", US Patent 5,649,023; D. Barbara and I. Kamel, "Method and Apparatus 

15 for Similarity Matching of Handwritten Data Objects", US Patent 5,710,916; D. 
Barbara and H. Korth, "Method and Apparatus for Storage and Retrieval of 
Handwritten Information", US Patent 5,524,240; D. Barbara and W. Aref, "Method 
for Indexing and Searching Handwritten Documents in a Database", US Patent 
5,553,284; R. Hull, D. Reynolds, and D. Gupter, "Scribble Matching", US Patent 

20 6,018,591; A. Poon, K.Weber, and T. Cass, "Searching and Matching 
Unrecognized Handwriting", US Patent 5,687,254; and, W. Aref and D. Barbara, 
"Trie Structure Based Method and Apparatus for Indexing and Searching 
Handwritten Databases with Dynamic Search Sequencing", US Patent 5,768,423. 

25 Other studies, J. Hollerbach, "An Oscillation Theory of Handwriting", Biological 
Cybernetics, pp. 139-156, 1981, and, Y. Singer and N. Tishby, "Dynamical 
Encoding of Cursive Handwriting", IEEE Conference on Computer Vision and 
Pattern Recognition, 1993, describe efforts to model the physical properties of 
handwriting for handwriting synthesis. 

30 

Disclosure Of Invention 
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The digital ink database searching techniques previously described are dependent 
on an ink query that is generated by the writer who authored the digital ink 
database. However, it would be beneficial if a digital ink database was searchable 
using other input mechanisms, for example, using a text query entered with a 
computer keyboard, or spoken and recognized by a voice recognition system. 
Alternatively, a third party may wish to search the digital ink database, either 
using his or her own handwriting, or using a text-based query. 

Ink database searching using handwriting feature synthesis allows a digital ink 
database to be searched using a text-based query. Using a writer-specific 
handwriting model derived from a handwriting recognition system or suitable 
training procedure, a text query is converted into feature vectors that are similar to 
the feature vectors that would have been extracted had the author of the digital ink 
database written the text query by hand. The feature vectors are then used to search 
the database, for example by using traditional techniques. This allows the 
searching of a digital ink database when the only input mechanism available is text 
entry, and can allow a person other than the author of the digital ink database to 
search the digital ink database. 

According to a broad form of the present invention, there is provided a method of 
searching a digital ink database using a text query, the method including: 

performing a search of a lexicon consisting of a table of letter sequences 
and associated feature vectors, and determining a sequence of lexicon entries 
which contain letter sequences that can be combined to produce the text query; 

obtaining a set of feature vectors corresponding to the sequence of lexicon 
entries; and, 

using the set of feature vectors to search the digital ink database. 

According to a particular embodiment of the present invention, the lexicon is part 
of a handwriting model obtained from the author of the digital ink database using 
handwriting recognition results. 
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In an embodiment of the present invention the handwriting model stores a mapping 
of letter sequences to feature vectors. Preferably, the lexicon includes multiple 
feature vectors for each letter sequence. Also preferably, if more than one 
sequence of lexicon entries is determined, then the sequence of lexicon entries 
5 having the least number of entries is used. 

In further embodiments, the text query is obtained by converting handwritten input 
into text using a handwriting recognition system, and/or, by converting voice input 
into text using a voice recognition system. 

10 

In a specific form of the present invention, the handwriting recognition results are 
obtained by: 

sampling the handwriting; 

smoothing the handwriting using a filter; 
15 performing slant correction; 

using a zone estimation algorithm to perform height normalisation; 

using feature extraction to perform segmentation into sub-strokes and 
generation of feature vectors; 

using feature reduction of the set of feature vectors; 
20 performing vector quantization to cluster the feature vectors and generate 

code word vectors; and, 

searching a dictionary for the most likely word from the generated text 

letters. 

25 According to a further broad form of the present invention, there is provided 
apparatus for searching a digital ink database using a query, the apparatus 
including: 

an input device for a user to input the query; 

a processor able to receive the query and communicate with the digital ink 
30 database; 

means to convert the query to a text query, if the query is not already a text 

query; 
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means for performing a search of a lexicon and determining a sequence of 
lexicon entries which contain letter sequences that can be combined to produce the 
text query; 

means for obtaining a set of feature vectors corresponding to the sequence 
5 of lexicon entries; 

means for searching the digital ink database using the set of feature vectors; 

and 

an output device to display the results of the search to the user. 

10 According to another aspect of an embodiment of the present invention, the text 
query is generated from handwriting using a different handwriting model than the 
handwriting model used to generate the feature vectors for searching. 

According to still a further broad form of the present invention, there is provided 
15 apparatus for searching a digital ink database using a query, the apparatus 
including: 

a store for storing the digital ink database; 
a processor, the processor being adapted to: 

perform a search of a lexicon and determine a sequence of lexicon 
20 entries which contain letter sequences that can be combined to produce the 

text query; 

obtain a set of feature vectors corresponding to the sequence of 
lexicon entries; 

use the set of feature vectors to search the digital ink database; 
25 wherein, the lexicon is part of a handwriting model obtained from the author of the 
digital ink database using handwriting recognition results. 

Brief Description Of Figures 

30 The present invention should become apparent from the following description, 
which is given by way of example only, of a preferred but non-limiting 
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embodiment thereof, described in connection with the accompanying figures, 
wherein: 

Figure 1 illustrates a processing system. 

Figure 2 illustrates an overview of the handwriting recognition method. 
5 Figure 3 illustrates the digital ink database searching method using feature 

synthesis. 

Figure 4 illustrates a third party ink database searching method. 
Figure 5 illustrates training and recognition / search phases. 
Figure 6 illustrates a method for handwriting recognition. 
10 Figure 7 illustrates an example for text recognition. 

Modes For Carrying Out The Invention 

The following modes are described as applied to the written description and 
15 appended claims in order to provide a more precise understanding of the subject 
matter of the present invention. 

L Preferred embodiment 

20 The present invention provides a method and apparatus for searching a digital ink 
database using a text query. In the figures, incorporated to illustrate the features of 
the present invention, like reference numerals are used to identify like parts 
throughout the figures. 

25 Embodiments of the present invention can be realised using a processing system an 
example of which is shown in figure 1. In particular, the processing system 10 
generally includes at least a processor 20, a memory 21, and an input device 22, 
such as a graphics tablet and/or keyboard, an output device 23, such as a display, 
coupled together via a bus 24 as shown. An external interface is also provided as 

30 shown at 25, for coupling the processing system to a digital ink database 11. 
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In use, the processing system 10 is adapted to allow data to be stored in and/or 
retrieved from the digital ink database 11. The processor 20 receives handwriting 
data, a text query, etc., via the input 22. From this, it will be appreciated that the 
processing system 10 may be any form of processing system or terminal such as a 
5 computer, a laptop, server, specialised hardware, or the like. 

Modelling Handwriting 

The writer-specific handwriting model describes the handwriting style of a 
particular user. Most user-adaptive handwriting recognition systems generate some 
10 kind of model to account for the stylistic variations between individual users. 
Generally, the purpose of these handwriting models is to map the feature vectors 
extracted from a set of input pen strokes into a set of letters that represent the 
recognized text. 

15 However, the feature synthesis approach to digital ink searching uses a writer- 
specific handwriting model to perform the inverse mapping. That is, the model is 
used to convert query text into a set of feature vectors that approximate the 
features that would have been extracted had the writer written the query text by 
hand. Figure 2 details the general steps in a handwritten text recognition system. 

20 The handwriting is sampled 26 and raw ink 27 is passed to a normalization step 28. 
The normalized ink 29 undergoes a segmentation step 30 and the resulting strokes 
31 are passed to a feature extraction step 32 which extracts the feature vectors 33. 
Classification step 34 is then performed using the handwriting model 35, which 
produces the primitives 36. Text recognition step 37 receives the primitives 36 and 

25 uses the language model 38 and/or handwriting model 35 to generate text 39 
corresponding to raw ink 27. 

To use feature synthesis for ink database searching, an additional step is required. 
The writer- specific handwriting model 35 is required to be modified to store 
30 information that allows text to be mapped to ink features. To perform this 
mapping, a table (called a lexicon) translating individual letters and groups of 
letters (i.e. sequences of letters) into features is included in the handwriting model. 
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After recognition has been performed, letters in the output text and the 
corresponding feature vectors used for recognition can be added to the lexicon. 

Groups of commonly co-occurring letters and their corresponding feature vectors 
5 are concatenated and added to the lexicon. This is desirable since handwriting 
exhibits co-articulation effects (where the writing of a letter is influenced by the 
shape of surrounding letters) and commonly written letters (e.g. "qu", "ed", and 
"ing") are likely to exhibit co-articulation. By storing the feature vectors of letter 
groups, a more accurate rendering of the query ink can be produced that considers 
10 the contextual effects of the stroke sequence. 

The lexicon should be able to store multiple feature vectors for each letter 
sequence. Since the same letters can be recognized many times, the lexicon should 
be able to select the feature vectors that best represents the letters. This can be 
15 done by storing all feature vector sequences output by the recognizer for each 
letter sequence, together with a count of the number of times that feature vector 
sequence has been seen for that letter sequence. The feature vector sequence with 
the highest count (i.e. the most frequently seen and thus most probable) for a letter 
sequence is then used during feature synthesis. 

20 

The choice of which letter groups should be stored can be based on character 
transition statistics (for example, as derived from a text corpus) where character 
sequences with a high probability of occurrence are stored (e.g. the probability of 
"ing" is much greater than the probability of "inx"). Alternatively, all possible 
25 letter groupings can be stored after recognition, with some kind of culling 
procedure executed when the table becomes too large (for example, least-recently 
used). 

Further improvements can be achieved by explicitly modeling word endings in the 
30 lexicon. Many writing styles exhibit poorly written characters at the end of words. 
This is particularly apparent in handwritten word suffixes such as "ing", "er", and 
"ed". To model this behavior, an end of word character is appended to the letter 
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sequence (e.g. "ing#") to indicate that this letter sequence models a word ending. 
During feature synthesis, these entries can only be used to complete a query word. 

Feature Synthesis 

5 To search a digital ink database, the text query is converted to a set of feature 
vectors by the feature synthesis procedure that uses the writer-specific handwriting 
model. These feature vectors are then used as the query term for searching the 
digital ink database. The ink database search can be performed using traditional 
ink matching techniques. Figure 3 describes this procedure. At step 40 text is input 

10 and the text 41 provided to feature synthesis step 42 which uses the handwriting 
model 35 to generate features 43. The features 43 are used in an ink searching step 
44 of the digital ink database 11. This produces ink matches 45. 

To perform feature synthesis, a search of the lexicon is performed to locate a 
15 sequence of lexicon entries that contain letter sequences that can be combined to 
produce the query text. The stored feature vectors are concatenated to produce the 
query feature vector. However, there may be a number of different combinations of 
lexicon entries that can be used to create the query text. It is assumed that using 
the minimum number of entries possible to produce the text will most accurately 
20 model the contextual effects. For example, assume the following entries exists in 
the lexicon: 



b 


bo 


bor 


borr 


ed 


ow 


rr 


rowed 



Table 1 . Example lexicon 



25 The word "borrowed" can then be created as (bo)(rr)(ow)(ed), (borr)(ow)(ed), or 
(bor)(rowed), with the final construction being most the desirable since it is 
composed from the fewest elements. 

A* Lexicon Search 
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For accurate results, the lexicon is expected to be very large, and the number of 
potential lexicon entry combinations for a word would be exponential. For queries 
containing long words, a complete enumeration of all permutations may not be 
practical. A modified A* search algorithm, see S. Russell and P. Norvig, Artificial 
5 Intelligence - A Modern Approach, Prentice Hall, 1995, can be used to search for a 
letter sequence s 9 where the path cost function g(s) is the number of lexicon entries 
used so far to create the text, and the estimated cost to the goal is: 

h(s) = 1, if lengthO) < lengthier);) 
10 = 0, otherwise 

This heuristic states that at least one additional lexicon entry is required to 
complete the letter sequence if the sequence has fewer letters than the query word. 
Nodes in the search tree are sorted by g(s) + h(s) (with lower scores being 
15 superior) and nodes with the same score are ordered by the number of letters in the 
sequence (with higher being superior). 

Note that h(s) is a monotonic and admissible heuristic (i.e. it never overestimates 
the cost of reaching the goal) so the search is guaranteed to find the optimal 
20 solution and is optimally efficient (i.e. expands the fewest nodes possible to find 
the optimal solution). Proof of this result is given in R. Dechter and J. Pearl, 
"Generalized Best-First Search Strategies and the Optimality of A*", Journal of 
the Association for Computing Machinery, 32(3), pp. 505-536, 1985. 

25 As an example of the above procedure, the search for the word "borrowed" is given 
below. Each row in the table represents a search node, with higher scoring nodes 
located at the top of the table: 



s 


g(s) 


h{s) 


g(s)+h(s) 


# Letters 


(borr) 


1 


1 


2 


4 


(bor) 


1 


1 


2 


3 


(bo) 


1 


1 


2 


2 
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The most promising node at the top of the table is expanded, resulting in: 



s 




h(s) 


g(s)+h(s) 


# Letters 


(bor) 


1 


1 


2 


3 


(bo) 


1 


1 


2 


2 


(borr)(ow) 


2 


1 


3 


6 



Again, the most promising node is expanded: 



s 




h(s) 


g( S )+h(s) 


# Letters 


(bor) (rowed) 


2 


0 


2 


8 


(bo) 


1 


1 


2 


2 


(borr)(ow) 


2 


1 


3 


6 



10 The top node is now a completed sequence, and no other node in the search can 
produce a better score, so this node is selected as the search result. 

II. Various embodiments 

15 HA. Feature Synthesis Without Handwriting Recognition 

Ink searching using feature synthesis can be performed without using a 
handwriting recognition system. The technique only requires the ability to build a 
lexicon of letter sequences and associated feature vectors to model the handwriting 
of the writer who authored the digital ink database. 

20 

If handwriting recognition results are not available for modelling, a training 
procedure can be used to generate the writer-specific lexicon. To do this, a user 
provides a sample of their handwriting by copying a specified training text, which 
is then used to build the lexicon. The training procedure is not required to perform 
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full handwriting recognition since the text represented by the handwriting is 
already known; rather, it simply needs to segment the input into characters and 
strokes, convert the strokes into features, and store the appropriate letter groups 
and associated feature vectors in the lexicon. 

The training text used to build the lexicon should be optimized to provide a 
balanced example set of individual letters and letter groups. That is, it should 
maximize the coverage of likely character unigrams, bigrams, and trigrams, see J. 
Pitrelli, J. Subrahmonia, M. Perrone, and K. Nathan, "Optimization of Training 
Texts for Writer-Dependent Handwriting Recognition", Advances in Handwriting 
Recognition, World Scientific Publishing, 1999, with an emphasis on letters and 
letter sequences that are most likely to be encountered. 

IIB. Different Features for Recognition and Ink Matching 

It is desirable that both the handwriting recognition system and the ink-matching 
algorithm use the same feature representations, since the features used to search 
the digital ink database are derived from the results of the handwriting recognition. 

However, it is possible to use different feature sets for the recognition and search 
provided the recognition features can be transformed into the search features. 
Some feature sets can allow a transformation from recognition features to search 
features to be learned from a set of training data. 

Alternatively, many feature sets can allow the regeneration of an approximation of 
the digital ink from the recognition features, from which the second feature set can 
be extracted. That is, the text query is converted to a set of feature vectors using 
feature synthesis, and the inverse transformation of the feature-extraction process 
is applied to the features to convert them into digital ink from which the search 
features are extracted. Care must be taken to ensure that this procedure does not 
introduce artifacts in the digital ink that may affect the search feature extraction 
(e.g. discontinuities in the generated ink may cause problems for some feature 
extraction techniques). 
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TIC. Third-Party Ink Searching 

A person can search the digital ink database of another writer by using handwriting 
recognition to convert their ink input into text, and using feature synthesis to 
convert the recognized text into features for ink searching. 

Figure 4 depicts this situation, with writer B searching the digital ink database 
authored by writer A. At step 46 ink is received from writer B and the strokes 47 
are passed to feature extraction step 48. The features 49 are extracted and 
recognition step 50 uses a language model 51 and writer B model 52 to generate 
corresponding text 53. This is used to build a lexicon for writer B at lexicon 
builder step 54. Letter groups and features are sent back to writer B model 52 to 
improve/update the model. The text 53 then undergoes feature synthesis step 42 
and a similar process as that described with reference to figure 3 is then followed 
to retrieve ink matches 45 that have been authored by writer A. 

HI. Further example 

The following example provides a more detailed outline of one embodiment of the 
present invention. This example is intended to be merely illustrative and not 
limiting of the scope of the present invention. 

This section describes in detail an implementation of ink database searching using 
feature synthesis. It is assumed that a number of preprocessing steps have been 
performed, including word and character segmentation, and baseline orientation 
normalization. Note that this is just one possible way to implement the technique; 
there are alternate methods available for each stage of the process. For example, 
there are many different segmentation schemes, feature sets, handwriting models, 
and recognition procedures that could also be used. 

The procedure for handwriting recognition and ink searching using feature 
synthesis requires a training phase and a recognition or searching phase. During 
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the training phase, a set of training data is converted to stroke features that are 
clustered into stroke primitives and used to create a model of the handwriting. For 
recognition and ink searching, this model is used to decode input ink or synthesize 
features for ink searching. This process is depicted in figure 5. Note that the same 
preprocessing, normalization, segmentation, and feature extraction procedure is 
used for training, recognition and searching. 

The handwriting recognition system maps input ink into a set of stroke code words 
that are used to search a dictionary for matching words. Figure 6 provides an 
overview of this process, with the individual steps described below in more detail. 

Smoothing 

The ink is sampled at a constant rate of 100 Hz. Research has shown that 
handwriting has a peak spectral density at around 5 Hz that declines to noise level 
at about 10 Hz, see H. Teulings and F. Maarse, "Digital Recording and Processing 
of Handwriting Movements", Human Movement Science, 3, pp. 193-217, 1984. 
Thus, a low-pass filter with a cutoff at 10 Hz will remove the high-frequency noise 
without affecting the relevant spectral components of the handwriting signal. 

A low-pass filter conforming to the above specifications can be produced by 
circularizing the point coordinates, performing an FFT to remove the high 
frequency components, and recreating the signal using an inverse-FFT. However, a 
simple weighted averaging filter works as effectively. To smooth a sequences of 
points {pi...p n ): 

k 

Pi = Jj<*jP»J 

j=-k 

where 

The filter width k and a smoothing coefficients are determined empirically. 



Slant Correction 
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Many handwriting styles do not have a vertical principal axis of the drawn letters 
(i.e. the letters are drawn at a consistent slant). Removing handwriting slant is a 
normalization that can improve the recognition of handwritten letters. In 
handwriting, down-strokes are considered the most stable and consistent strokes, 
5 and thus are useful for detecting handwritten slant. 

To detect handwritten slant, a weighted-average direction of the down strokes in 
points {pi...p n } is estimated: 

n 

slant = — 

n 

2*. 

i=2 

10 where 

at = angle (pupui) 
di = lP/-/Vi|| if ai <fl/<a 2 

= 0 otherwise 

15 Angles ai and a 2 define which stroke segments are considered as parts of a down 
stroke and are empirically set at 40° and 140° respectively (with 90° representing a 
vertical line). If the estimated slant deviates more than a certain threshold from the 
vertical, slant is removed using a shear transformation: 

(^max-^)xtan(90 o -^n0 

20 Xj — 

y max y min 

where y mi „ and y max represents the top and bottom of the bounding rectangle of the 
ink. 

25 Zone Estimation 

Zone estimation is used to normalize the height of the input ink. English letters 
exhibit three zones - the middle zone (corresponding to the height of letters such 
as 'a', 'c\ 'e', etc.), and the upper and lower zones that contain the ascenders and 
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descenders in letters such as *b\ c d\ 'g' ? and 'j\ 

Zone estimation is performed using a horizontal histogram of ink density. That is, 
the number of ink crossings is determined for an equally spaced series of 

5 horizontal lines passing through the bounding rectangle of the ink. The central 
peak of the histogram is found, as are the two points on either side of the 
histogram where the ink density drops below a certain fraction of the central peak 
height. These two points are selected as the upper and lower bounds of the middle 
zone. The upper and lower zones are defined as the space between the middle zone 

10 and the vertical extremum of the bounding rectangle. 

Feature Extraction 

The ink is segmented into sub-strokes at extrema in the vertical direction (i.e. at 
local maxima and minima of the Y coordinates). For segmentation to occur, the 
15 lengths of the two sub-strokes produced by splitting the stroke at the selected 
segmentation point must exceed a pre-calculated minimum distance (set at half the 
height of the estimated middle zone). 

The segmented sub-strokes are then re-sampled to contain a constant n number of 
20 points located equi distantly along the stroke trajectory. A feature vector is then 
created for the sub-stroke by normalizing the coordinates: 

f _ X i ~ X min 

Xi ~ h 

y middle 
h 

X minimum of sub-stroke bounding rectangle 
Y coordinate of the top of the middle zone 
the height of the middle zone (i.e. ybase-y middle) 




25 where 

y middle = 

h 



30 The feature vector is then created from the normalized coordinates as f f = 
{x'i,y'i,...,x'n,y'n}. 
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Feature Reduction 

The resulting vectors describe the sub-strokes using a large number of highly 
correlated features (clearly, the coordinates of point p t are dependent on point p U j 
and so on). To lower the dimensionality of the vectors to m (where m < 2ri), the 
Karhunen-Loeve transform (see Principal Component Analysis - R. Duda, P. Hart, 
and D. Stork, Pattern Classification, Second Edition, John Wiley & Sons, Inc., pp. 
569-570, 2001) is used. This procedure projects the higher-dimensionality features 
into a lower dimension using linear mapping that is optimal in a least-squares 
sense. 

To do this, the covariance (autocorrelation) matrix for the set of all training feature 
vectors X={fi,...,f w } is calculated using: 

c = — !— x T x 

m — l 

The eigenvectors and eigenvalues for this matrix are found (using the tridiagonal 
QL implicit algorithm, see W. Press, B. Flannery, S. Teukolsky, and W. Vetterling, 
W. T., Numerical Recipes in C, Cambridge: Cambridge University Press, 1988), 
and the eigenvectors corresponding to the largest n eigenvalues are used to form 
the PCA matrix Z. The feature vectors are then multiplied by this matrix to 
transform the features into a new feature space with an orthogonal uncorrelated 
axis: 

Vector Quantization 

The transformed feature vectors are then clustered using a Kohonen Self- 
Organizing Feature Map (SOFM), see T. Kohonen, "Self-Organized Formation of 
Topologically Correct Feature Maps", Biological Cybernetics, 43, pp. 59-69, 1982. 
This technique uses an unsupervised learning procedure to cluster the input vectors 
such that distance and proximity relationships between the vectors is maintained as 
far as possible. The SOFM used has a 2-dimensional structure, with visually 
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similar code words (i.e. clusters) located near each other. As a result, the distance 
between two code words can be easily calculated using some distance measure 
(e.g. Euclidean distance) between the code word values. 

5 SOFM training is performed iteratively using a simple two-layer neural network 
that is initialized with random weights. The best matching output neuron for a 
normalized input training vector x is found using minimum Euclidean distance: 

o(x) = argmin||x- w f || 

10 where represents the weight vector of output node i. The weights of the node 
with the highest activation value and those nodes surrounding it (as determined by 
a neighborhood function A) are updated using: 

W t (t + 1) = W,(f) + T1(0[X - W,(*)] 

15 where r| is a learning rate function and r\ and A are typically varied over time t. 
Training continues until there is no noticeable change to the neuron weights during 
an iteration of the training set. 

To convert a sequence of sub-strokes to a code word vector, each sub-stroke 
20 feature vector is quantized using the trained SOFM and appended to the code word 
vector. A feature vector is quantized into a code word by selecting the largest 
activation value for the output neurons of the SOFM codebook: 

T 

q = argmaxx w f 



25 Handwriting Model 

The handwriting model stores a mapping of stroke code word vectors to letters (for 
text recognition) and the reverse mapping of letter groups to code word vectors 
(for feature synthesis). To build the handwriting model, each letter in the training 
data is converted to a code word vector that is stored in a table, along with the 

30 corresponding letter. 
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Note that a specific code word sequence may map to a number of letters (e.g. 
poorly-drawn 6 u' may map to the same feature vectors as a 6 v') and individual 
letters may be mapped to by a number of code word vectors. By maintaining a 
count of the number of times the code word vector represented a specific letter, the 
5 probability that the vector represents the letter x t can be calculated given n lexicon 
entries for letter x t \ 

M 

where a is the count of the number of times the code word vector has been 
10 encountered representing the letter x t . The following is an example entry from the 
table that represents the hypothetical code word vector {3,4}: 





ci 


Piped 


'u' 


120 


0.54 


'v' 


91 


0.41 


'r' 


12 


0.05 



Table 2. Example code word vector entry 

This table indicates that if the code word sequence {3,4} is encountered in the 
15 input, there is a probability of .54 that it represents the letter 'u\ a probability .41 
that it represents an *v\ and a probability of .05 that it represents an 'r\ 

The reverse mapping table is produced in a similar way, storing code word vectors 
associated with letters and groups of letters. 

20 

Recognition 

To perform handwriting recognition, the input ink is processed as described above, 
and the resulting code word vectors are used to search the handwriting model to 
produce letter hypotheses. A best-first search strategy is used to search a 
25 dictionary for the most likely word given the derived letter probabilities. The 
process is depicted below in Figure 7. 
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Ink Searching 

Ink searching is performed by using the handwriting model generated during 
handwriting recognition to map the input query text into a sequence of code words. 

5 This code word vector is used to search the digital ink database using an elastic 
matching technique. A full description of a similar ink-matching technique is given 
in D. Lopresti and A.Tomkins, "Temporal-Domain Matching of Hand-Drawn 
Pictorial Queries", Handwriting and Drawing Research: Basic and Applied Issues, 
IOS Press, pp. 387-401, 1996. The resulting queries are then ordered by similarity 

10 and presented to the user. 

Thus, there has been provided in accordance with the present invention, a method 
and apparatus for searching a digital ink database using a text query which satisfies 
the advantages set forth above. 

15 

The invention may also be said broadly to consist in the parts, elements and 
features referred to or indicated in the specification of the application, individually 
or collectively, in any or all combinations of two or more of said parts, elements or 
features, and where specific integers are mentioned herein which have known 
20 equivalents in the art to which the invention relates, such known equivalents are 
deemed to be incorporated herein as if individually set forth. 

Although the preferred embodiment has been described in detail, it should be 
understood that various changes, substitutions, and alterations can be made herein 
25 by one of ordinary skill in the art without departing from the scope of the present 
invention as hereinbefore described and as hereinafter claimed. 
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CLAIMS 

1. A method of searching a digital ink database using a text query, the method 
5 including: 

performing a search of a lexicon consisting of a table of letter sequences 
and associated feature vectors, and determining a sequence of lexicon entries 
which contain letter sequences that can be combined to produce the text query; 

obtaining a set of feature vectors corresponding to the sequence of lexicon 
10 entries; and, 

using the set of feature vectors to search the digital ink database. 

2. The method as claimed in claim 1, wherein the lexicon is part of a 
handwriting model obtained from the author of the digital ink database using 

15 handwriting recognition results. 

3. The method as claimed in either claim 1 or claim 2, wherein the handwriting 
model stores a mapping of letter sequences to feature vectors. 

20 4. The method as claimed in any one of the claims 1 to 3, wherein the lexicon 
includes multiple feature vectors for each letter sequence. 

5. The method as claimed in any one of the claims 1 to 4, wherein the lexicon 
includes an end of word character to indicate word endings. 

25 

6. The method as claimed in any one of the claims 1 to 5, wherein the 
handwriting recognition results used to generate the lexicon are derived from a 
training text. 

30 7. The method as claimed in any one of the claims 1 to 6, wherein if more than 
one sequence of lexicon entries is determined, then the sequence of lexicon entries 
having the least number of entries is used. 



WO 03/034276 



23 



PCT/AU02/01395 



8. The method as claimed in any one of the claims 1 to 7, wherein a search 
algorithm is used to search for a letter sequence s. 

9. The method as claimed in any one of the claims 1 to 8, wherein the text 
query is obtained by converting handwritten input into text using a handwriting 
recognition system. 

10. The method as claimed in any one of the claims 1 to 8, wherein the text 
query is obtained by converting voice input into text using a voice recognition 
system. 

11. The method as claimed in any one of the claims 1 to 10, wherein the 
handwriting recognition results are obtained by: 

sampling the handwriting; 

smoothing the handwriting using a filter; 

performing slant correction; 

using a zone estimation algorithm to perform height normalisation; 
using feature extraction to perform segmentation into sub-strokes and 
generation of feature vectors; 

using feature reduction of the set of feature vectors; 

performing vector quantization to cluster the feature vectors and generate 
code word vectors; and, 

searching a dictionary for the most likely word from the generated text 

letters. 

12. Apparatus for searching a digital ink database using a query, the apparatus 
including: 

an input device for a user to input the query; 

a processor able to receive the query and communicate with the digital ink 
database; 
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means to convert the query to a text query, if the query is not already a text 

query; 

means for performing a search of a lexicon and determining a sequence of 
lexicon entries which contain letter sequences that can be combined to produce the 
text query; 

means for obtaining a set of feature vectors corresponding to the sequence 
of lexicon entries; 

means for searching the digital ink database using the set of feature vectors; 

and 

an output device to display the results of the search to the user. 

13. The apparatus as claimed in claim 12, wherein the lexicon is part of a 
handwriting model obtained from the author of the digital ink database using 
handwriting recognition results. 

14. The apparatus as claimed in either claim 12 or claim 13 , wherein the text 
query is generated from handwriting using a different handwriting model than the 
handwriting model used to generate the feature vectors for searching. 

15. The apparatus as claimed in any one of the claims 12 to 14, wherein the 
lexicon is produced using a training text. 

16. The apparatus as claimed in any one of the claims 12 to 15, wherein the text 
query is generated from a voice query by a voice recognition system. 

17. Apparatus for searching a digital ink database using a query, the apparatus 
including: 

a store for storing the digital ink database; 
a processor, the processor being adapted to: 

perform a search of a lexicon and determine a sequence of lexicon 
entries which contain letter sequences that can be combined to produce the 
text query; 



WO 03/034276 



25 



PCT/AU02/01395 



obtain a set of feature vectors corresponding to the sequence of 
lexicon entries; 

use the set of feature vectors to search the digital ink database; 
wherein, the lexicon is part of a handwriting model obtained from the author of the 
5 digital ink database using handwriting recognition results. 
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